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ABSTRACT 


Near real-time tele-operated driving on the lunar surface 
remains constrained by bandwidth and signal latency de- 
spite the Moon’s relative proximity. 

As part of our work within NASA’s Human-Robotic Sys- 
tems Project (HRS), we have developed a stand-alone 
modular LIDAR based safeguarded tele-operation system 
of hardware, middleware, navigation software and user 
interface. The system has been installed and tested on two 
distinct NASA rovers-JSC’s Centaur2 lunar rover pro- 
totype and ARC’s KRex research rover-and tested over 
several kilometers of tele-operated driving at average sus- 
tained speeds of 0.15 - 0.25 m/s around rocks, slopes and 
simulated lunar craters using a deliberately constrained 
telemetry link. 


The navigation system builds onboard terrain and hazard 
maps, returning highest priority sections to the off-board 
operator as permitted by bandwidth availability. It also 
analyzes hazard maps onboard and can stop the vehicle 
prior to contacting hazards. It is robust to severe pose 
errors and uses a novel scan alignment algorithm to com- 
pensate for attitude and elevation errors. 


Key words: navigation; lunar tele-operation, Centaur2, 
hazard detection, RAPID, NASA telemetry standard, 
point cloud alignment, LIDAR, roughness estimation. 



Figure 1: The Centaur2 rover at NASA JSC’s Rocky ard 
test site. Note the modular navigation mast installed on 
the right 



Figure 2: NASA ARC’s KRex research rover 



1. INTRODUCTION 


2. HARDWARE 


Near real-time tele-operated driving on the lunar surface 
remains constrained by bandwidth and signal latency. 
Proposed robotic lunar science missions [1] require ve- 
hicles to drive 3-5 km in 4-7 days to accomplish their 
objectives. NASA’s Human Architecture Team recom- 
mends a maximum rate of 384 kbps for downlink and 10 
kbps rate for uplink [2] . 

This work documents our effort to demonstrate a mod- 
ular system for tele-operating a lunar rover over a kilo- 
meter subject to a 1 Mbps bandwidth constraint for un- 
compressed data. The system consists of sensors, com- 
putation, middleware, navigation software and a UI. It is 
deployed on two distinct NASA robots - JSC’s Centaur2 
Lunar rover prototype (Figure 1) and ARC’S KRex re- 
search platform (Figure 2). 

This work builds on our earlier LIDAR navigation stack 

[3] , first developed to navigate CMU’s SCARAB rover 

[4] in complete darkness, and later demonstrated on 
NASA’s Lunar Electric Rover (now called the Space Ex- 
ploration Vehicle or SEV) during the 2011 DRATS simu- 
lated lunar operations test [5]. The algorithms used have 
direct heritage from Simmons et al classic RANGER ar- 
chitecture [6], derivatives of which have since been used 
by stereo- vision based planetary rovers, and the work of 
Thrun et al for the DARPA Grand Challenge [7, 8]. 

The subsequent sections detail the navigation sensor and 
computation stack and its integration to the Centaur2 
platform, the RAPID middleware and telemetry standard 
developed in conjunction with this project, the mapping 
algorithms, and user interfaces. Preliminary results from 
a June 2012 demonstration of kilometer scale driving un- 
der the constrained bandwidth are also presented. 
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Figure 3: Navigation sensor mast, installed on Centaur2 
and KRex 
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Figure 4: Centaur2 hardware and networking 


The Centaur2 navigation "sensor mast" (Figure 3) is 
a standalone system that can be easily ported between 
rovers of similar size. It includes terrain and pose sen- 
sors plus computation in a single package. 

The Velodyne HDL-32E LIDAR is the primary terrain 
mapping and navigation hazard detection sensor. Thirty- 
two beams provide simultaneous range measurements 
from -30° to +10° in elevation above and below the sen- 
sor, with a complete 360° scan around the rover 10 times 
per second. We subsample the data in time. 

Forward-looking stereo cameras (Table 1) provide denser 
range measurements ahead, and in particular include the 
Velodyne blind spot immediately in front of the rover 
caused by the minimum elevation angle of the lidar. This 
additional coverage is useful for looking down over cliffs 
or on steep slopes. 

The fan-less navigation computer (Table 2) has separate 
network connections to the LIDAR and Gigabit Ethernet 
cameras to isolate their traffic from the main rover LAN 
(Figure 4). On Centaur2, additional computers provide 
pose estimation (Table 3), executive and motion control 
services. Frequent (every 5 minutes) NTP synchroniza- 
tion is required between the navigation and pose esti- 
mation computers. A Tropos wireless network links the 
rover LAN to ground data systems on NASA’s network 
via a specialized router configured to enforce a 1 Mbps 
connection limit and (optionally) introduce a user speci- 
fied communications latency. 

Table 1 : Stereo cameras 


Stereo baseline 30cm 

Lens field of view 82° x 67° 

Lens focal length 3.5mm (C-mount) 

Camera sensor color 1388x1 03 8 pixels 

Camera interface Gigabit Ethernet 



Table 2: Navigation computer 


Processor 

Intel Core i7-620M @ 2.66GHz 

Storage 

120GB SSD 

Memory 

4GB DDR3 

Cooling 

passive fan-less 

Serial ports 

3 x RS-232 + 1 x RS-232/422/485 

Network ports 

5 x Gigabit Ethernet 

Physical 

240x76x195mm - 2.7 kg 


Table 3: Pose estimation systems and estimated accura- 
cies 



KRex 

Centaur2 

Position 

Novatel OEM 
4DGPS 

Topcon GPS 


5 cm 

3.5m horizon- 
tal 

5m vertical 

Orientation 

HG1700 FOG 

Crossbow 

VG700 


0.1° roll, pitch 

2.5° roll, pitch 


0.3° yaw 

> 5° yaw 
Estimated 
from GPS 


3. ARCHITECTURE AND MIDDLEWARE 


The safe-guarded tele-operations system is designed at 
its core as a distributed system of decoupled modules 
running on multiple machines, both onboard and off- 
board. Communications between modules and between 
robot and ground-control uses NASA’s inter-center Robot 
API Delegate (RAPID) interfaces and message specifica- 
tions [9], built on the Data Distribution Service (DDS) 
open standard for data-centric publish-subscribe architec- 
tures [10], and developed in partnership with this project. 

RAPID/DDS accommodates the large differences in 
bandwidth and latency of the on-robot versus robot- 
ground communication links, identified by unique DDS 
domain ID’s. The Quality of Service (QoS) parameters 
for each domain are uniquely adjusted to the characteris- 
tics of each link. A key parameter is reliability. Certain 
data products (e.g. pose) are quickly stale, and bandwidth 
should not be wasted on protocols to ensure delivery of 
all pose data offboard. Conversely, maps data must be 
reliably sent from the mapping module to the path evalu- 
ation module in a timely manner. 

To date, we have not taken full advantage of the DDS 
middleware capabilities; for simplicity some data streams 
(maps) are not yet split into on-board and off-board 
streams. Additional DDS features for future investigation 
include lossless data compression, header compression, 
and rate limiting. 

The mapping and path-analysis modules are built within 
our Service Oriented Robotics Architecture (SORA) 
framework [11], consisting of a number of distinct mod- 


ules that subscribe to common data products (sensor mea- 
surements, pose estimates) via the RAPID/DDS commu- 
nications layer, and respond to remote procedure calls 
from other modules or the User Interface (UI). 

The inherently distributed nature of SORA and the 
RAPID message specifications reduced the complexity of 
integrating the navigation hardware and software to Cen- 
taur2, which has its own distinct telemetry, pose estima- 
tion, sequencing, and motion control subsystems running 
on different operating systems (the navigation stack runs 
on Linux, the pose estimation and sequencing on Win- 
dows, and motion controllers on VxWorks). Through im- 
plementation of a bridge to the commonly agreed RAPID 
API the two architectures could be tied together. 

The power of the RAPID and SORA based approach is 
nicely illustrated by the fact that the system was devel- 
oped as an inter-center project, with the navigation stack 
being deployed and tested on two different NASA robot 
platforms simultaneously (Centaur2 [12] and KRex [13]), 
due to availability of the robotic platforms at the different 
NASA centers. Centaur2 and KRex feature very different 
control architectures and have different hardware charac- 
teristics. 


4. HAZARD DETECTION AND MAPPING 


The hazard detection and mapping subsystem, shown 
in Figure 5, combines range measurements r^(t) with 
the corresponding rover pose estimate P{t) (interpolated 
from the previous and subsequent pose estimation system 
outputs) to compute the point cloud Xi(t). A small pose 
correction dP(t ) is computed to align the point cloud 
with the current topography map. The aligned points 
Xi ( t ) update their corresponding cells in the statistics ac- 
cumulator map E while dP(t ) is retained for future up- 
dates. 

The statistics accumulator map E, represented as a 
scrolling rectangular array for constant time determina- 
tion of point-cell correspondences, tracks the correlation 
moments, min and max heights and other (minimal) in- 
formation needed to robustly compute the topography 
and height difference maps. Accumulated statistics from 
the past are progressively de- weighted in favor of current 
data to account for changes in the environment and accu- 
mulated pose errors. For details see [3]. 

Height difference and subsampled topography maps are 
computed twice per second from the accumulator map, 
and merged with a user-specified map to create the com- 
bined navigation map, consisting of travers ability, cer- 
tainty and roughness values within [0,1] (equation 1). 
Traversability can be interpreted as an inverse cell traver- 
sal cost for path evaluation and planning purposes. Haz- 
ards that must be avoided have zero traversability (oo 
cost). Cells with certainty less than 0.5 are considered 
unexplored, and also treated as hazards for the purposes 



of local path evaluation. Nearby unexplored regions are 
due to occlusions, steep drop offs or holes in the ground. 

trav =min(tr s i ope , tr height diff ) (1) 

tv Q =ls(6, 6 cau ti on , Ohazard) (2) 

( 0 x > b 

ls(x,a,b)=< 1 x<a (3) 

[ (b — x)/{b — a) o.w. 

Certainty is similarly computed based on the accumu- 
lated weight of points assigned to each cell in the accu- 
mulator map. This works when the vehicle is in motion 
and the points cover the full cell area, but breaks down 
when stationary and the same points are repeatedly added 
to a cell. 

The combined map is broken into smaller map tiles that 
are individually exported over the DDS telemetry channel 
and individually displayed in the UI. Only map tiles that 
change are exported, and they are prioritized by proxim- 
ity to the rover’s current position. Lower priority tiles are 
discarded when bandwidth is constrained. The UI dis- 
plays all tiles received, including those corresponding to 
areas left behind the rover and no longer being actively 
mapped. 

Path checking is efficiently (albeit conservatively) done 
on a 2D C- space map, wherein hazards and unexplored 
areas are expanded by the robot radius. Our map repre- 
sentation is such that we can do this with the OpenCV 
erode function. 

Pose computed from GPS, or infrequent landmark obser- 
vations, contains discontinuities that corrupt the maps, 
challenge the point cloud alignment, and lead to appar- 
ent changes in hazard locations with respect to the rover. 
To prevent this, relative position (from odometry only) 
is used in the mapping and collision checking modules. 
This approach is based on an assumption that pose in- 
accuracies accumulate sufficiently slowly to avoid major 
map distortions. 

The rover maintains the transform between its relative 
(odometry only) and global (using GPS) pose estimates. 
Each exported map tile is tagged with its global pose so 
that the UI will display it correctly relative to prior maps 
and overhead images of the site. This results in the tiles 
shifting slightly with respect to each other as the two dif- 
ferent pose estimates diverge, but the tiles remain self- 
consistent and locally consistent. 


4.1. Map alignment 

The map alignment algorithm, illustrated in Figure 6, 
uses Point-To-Plane ICP with small angle trigonometry 
approximations to efficiently compute the rotation and 
translation that minimizes the sum of squared normal 
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Figure 5: Mapping Algorithm. White map cells 

have no data (certainty < 0.5), green indicates drivable 
(traversability =1), red hazards (travers ability = 0), and 
yellow caution (traversability in (0,1) 


distances between points and their corresponding planes 
[14]. The algorithm proceeds as follows: 
for all points x t in point cloud do 
get corresponding map cell k 
if ACCEPT (x,k) then 

add cell center 6k, normal rik, and normal std 
dev (j k to lists {6k}, { n k }, {a k } 

end if 
end for 

COMPUTE 

dP ={dz , droll, dpitch) T (4) 

—argmin J2(dP * (x k - P) + P - o k ) ■ fi k /o k 

k 

( 5 ) 


The correspondence acceptance test above requires that 
the cell correspond to planar terrain, the point be close 
to the cell center, and time between point acquisition and 
last update of cell statistics be limited. Cells are consid- 
ered planar if the standard deviation (in the normal direc- 
tion) of all prior points added to the cell is a fraction of 
the cell width. 

Only one iteration is required, since point-cell correspon- 
dences barely change after application of dP. As such, 
the algorithm is fast enough to apply to every input scan. 
Each time dP is accumulated and applied to the next 
point cloud prior to alignment. 

4.2. Height Difference Maps 

Rocks higher than the vehicle ground clearance (30cm 
for Centaur2) are hazards. A map cell is thus considered 
untraversable if two points within e horizontal distance 




Figure 6: Point cloud alignment to terrain map 


of each other differ in height by by more than a threshold 
S. Naively applying this test results in an unacceptable 
number of false positives due to pose errors, even after 
applying map registration. 

Thrun et al [8] developed a statistical test for hazardous 
height differences that is robust (up to a point) to pose 
noise. Essentially, given two points (X k , Z k ,Z k ) T and 
(X^ Z^ Zi} n ) T acquired at times k and m, then 


Pr(\Zi-Zf\>6)>n 

(6) 

if and only if 


1 Z{ — Zf \ — 5 > T 

(7) 

Where T is of form 


4 

II 

© 

1 

+ 

(8) 


and depends on the corresponding range measurements 
and the statistics of the pose noise. 


To compute whether a cell passes the obstacle test (6) it is 
sufficient to track the minimum and maximum heights of 
corresponding points, along with their associated range 
measurements and acquistion times. 

This test has the drawback of disregarding obstacles just 
below the detection threshold. Tracking the actual height 
differences would enable us to flag cautionary obstacles, 
and generalize the map for use with different bottom 
clearances (Centaur2 can adjust this). We invert (7) to 
track a statistically robust lower bound Si on the height 
differences at each cell i (or between cells e apart): 

Si = max(5° ld ,max(\Z l k — Zf \ — T)) (9) 

This can be reduced (approximately) on slopes 6 by 
ecos(0 ), the height difference due solely to terrain incli- 
nation. 


5. USER INTERFACE 


The user interface draws components from two projects. 
The first, PIGI (Predictive, Interactive Graphical Inter- 


face) provides interfaces for teleoperating robots over 
time-delayed and limited bandwidth links. Second, 
VERVE (Visual Environment for Remote & Virtual 
Exploration), provides 3D displays to visualize robot 
telemetry in real time. Both projects are part of the NASA 
Ensemble platform, a component-based plugin architec- 
ture based on the Eclipse RCP (Rich Client Platform) and 
implemented in Java. 

The Centaur2 driving interface is provided by PIGI, a 
suite of tools designed to keep the human in the loop as 
much as possible while taking advantage of short-term 
robot autonomy [15]. A typical command cycle involves 
the operator placing a waypoint, then sending that com- 
mand to a behavioral simulator which predicts the robot’s 
drive path. If the operator is satisfied with the path, the 
command is issued to the robot’s task queue, and the op- 
erator begins planning the next waypoint. PIGI integrates 
with VERVE to display the waypoints and predicted drive 
paths in the 3D view, and also includes user interface ele- 
ments for manipulating the robot’s task queues and view- 
ing robot status information. 

The VERVE 3D components are designed for high fi- 
delity visualization of multi-robot operating scenarios. 
VERVE supports large scale terrain rendering through the 
use of geometry [16] and texture clipmaps [17] for op- 
erations in outdoor environments. Multiple sets of tiled 
DEMs (Digital Elevation Model) and orthorectified satel- 
lite imagery may be composited at runtime to provide an 
unbounded base map. Robots avatars are placed in the 
scene based on the latest pose estimate and articulated 
by joint telemetry. An extensible library of visualizations 
is available for the user to display raw sensor data, de- 
rived data products, and other telemetry. For the purposes 
of the safe-guarded driving task, visualizations for navi- 
gation maps and terrain analysis trajectories were most 
commonly used. 
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Figure 7: Navigation map rendering modes. Clockwise 
from upper left: traversability map, c-space map, slope 
map, roughness map 


The RAPID navigation map data structure consists of a 
transform, a tile id, and an array of labeled layers which 
contain a grid of arbitrary data. Each layer is allowed 



an independent subsampling factor to reduce map size on 
the wire. If a layer labeled as "height" is present in the 
map it will be used to construct the map tile mesh, oth- 
erwise, the mesh is constructed from either height values 
obtained from the base map, or from height layers ob- 
tained from another navigation map topic. Data from all 
other layers is uploaded directly to the GPU for process- 
ing. The user can choose how to display the navigation 
maps by selecting one of the GLSL shaders for differ- 
ent rendering modes, and widgets are provided to inter- 
actively modify shader parameters such as thresholds and 
color ramps. The operator typically uses the travers ability 
shader while driving, although the roughness, slope, and 
c-space shaders (Figure 7) are helpful for gaining insight 
about the terrain when the safe-guarded driving module 
halts the vehicle. 

The terrain analysis trajectory visualization gives the op- 
erator insight into how the safe-guarded driving module 
is perceiving the terrain. When a drive command is issued 
to the robot, the planned drive path is sampled at regular 
intervals and tested against the c-space navigation map to 
classify points along the path as safe, hazardous, or un- 
known. The result is visualized as a color coded trajec- 
tory with arrow icons placed at regular intervals to indi- 
cate orientation of the vehicle at along the path. Flashing 
icons indicate that the path has intersected a hazardous or 
unknown region within stopping distance of the vehicle. 


6. RESULTS 


In June 2012 we tested the tele-operation system at the 
JSC Rocky ard (Figure 1). Four separate drives were ac- 
complished by the operator (Figure 8) in a separate build- 
ing, over a 1 Mbps bandwidth wireless link. At each com- 
mand cycle the operator would review the maps and se- 
lect the next waypoint. Multiple waypoints can be queued 
up, and execution may be interrupted at any time. An E- 
stop operator behind the robot was on hand to stop the 
vehicle if at any point the rover was in danger. No guid- 
ance was given to the driver by the E-stop operator. 

Table 4 summarizes the drive results. No attempt has 
yet been made to systematically quantify the effects of 
changing system parameters (bandwidth, latency, detec- 
tion sensitivity) on drive performance. Nonetheless we 
gained some qualitative insight to system performance. 
The one E-stop event occurred when the system was 
started up with a hazardous rock right in front of the vehi- 
cle, below the LIDAR field of view (the stereo hardware 
is in place to rectify this but is not yet tied in). 

A demonstration highlight was summiting "Mt Kosmo", 
a 15m high hill with steep drop offs on 3 sides (Figure 
9). The operator had to turn off the safe-guarded driving 
mode to get past some vegetation clumps that registered 
as obstacles (Figure 10, 11) 

Craters (Figure 12) were hard to see in the camera images 
but showed up clearly in the maps (Figure 13). Crater in- 




Figure 9: Centaur2 at summit of Mt Kosmo 


teriors remained occluded until the rover was well inside 
(requiring the disabling of safeguarded driving). 

Hazardous rocks are detected (Figure 14), albeit doing so 
reliably currently requires turning off the slope compen- 
sation detailed earlier. Slope compensation is effective at 
reducing false positives but also reduces rock detection in 
dense rock fields (Figure 15). 


Table 4: June 2012 tele-operation demonstration results 


Test# 

1 

2 

3 

4 

Distance 

496m 

435m 

220 m 

253m 

# cmd cycles 



13 

10 

Avg cycle dist 



17m 

25m 

mean speed 

0.24m/s 

0.18 m/s 

0.15m/s 

0.14m/s 

E-Stops 

1 




False Stops 



2 


True Stops 




3 




Figure 10: Navigation UI as Centaur2 ascended Mt 
Kosmo. Note the small false obstacles due to grass, 
which the operator over-rode by turning off the safe- 
guarded driving mode. 


Figure 13: Driving around craters. Significant occluded 
areas inside craters resulted in path vetoes. Going in- 
side requires the operator to override safe driving mode. 
Craters show up clearly on the maps, even when they are 
hard to see in the driving images. 



Figure 1 1 : Oblique view of navigation map returned by 
Centaur2 as it ascended Mt Kosmo. 



Figure 12: Centaur2 in crater. 



Figure 14: Rock field map with slope correction disabled, 
showing significantly better rock detection at the cost of 
greater sensitivity to vegetation on surrounding slopes. 



Figure 15: Map of rock field (top) and E-Stop operator 
walking in front of rover (red trail) causing the robot to 
stop. 








7. CONCLUSIONS 


We have demonstrated a modular tele-operation system 
on two distinct NASA rovers with very different pose es- 
timation systems. The novel map alignment algorithm is 
essential for the creation of consistent terrain maps, and 
enables the use of pose sensors of modest accuracy. 

This project remains under active development at this 
time. Specific next steps include using the stereo vision 
system to detect hazards immediately in front of and be- 
low the rover (the LIDAR blind spot), visual odometry 
for improved local pose estimation, and aligning the LI- 
DAR point clouds with the prior basemap DEM for a 
global pose correction that might eliminate the need for 
GPS entirely (if the basemap is of sufficient resolution). 
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